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FOREWORD 


This  report  documents  the  methodology  and  lessons  learned  in  the  development  of  the 
Innovative  Tools  and  Techniques  for  Brigade  and  Below  Staff  Training  II  -  Battle  Staff  Training 
System  H  (ITTBBST-BSTS  II).  The  ITTBBST-BSTS II  project  was  developed  as  part  of  the 
U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences’  ongoing  research  for  the 
Force  XXI  Training  Program  to  establish  innovative  methodologies  for  training  combined  arms 
forces.  The  Armored  Forces  Research  Unit  has  conducted  previous  research  to  develop  text  and 
computer-based  courses  of  instruction  to  train  commanders  and  staffs  in  their  fundamental  combat 
skills  using  scenario-based  training  support  packages  (TSPs).  This  project  was  initiated  to 
eliminate  the  requirement  for  text-based  materials  in  training  staff  officers. 

The  ITTBBST-BSTS  II  research  effort  built  on  the  library  of  TSPs  that  train  selected  staff 
officers  and  commanders  in  current  Army  battlefield  doctrine,  allowing  them  to  practice  using  that 
doctrine.  One  component  of  the  project  converted  the  existing  Brigade  Common  Core  course 
from  a  text  and  computer-based  TSP  to  a  100%  computer-based  training  (CBT)  course  of 
instruction.  The  second  component  developed  the  means  to  train  government  personnel  in  the 
process  of  updating  and  maintaining  the  existing  library  of  BSTS  courseware. 

Results  of  this  research  have  been  provided  to  the  Directorate  of  Training  and  Doctrine 
Development’s  Force  XXI  Training  Program  at  the  US  Army  Armor  Center  and  serve  as  an 
excellent  model  for  future  efforts  to  develop  and  implement  individual  computer-based 
instructional  courseware. 


mjA  M.  SIMUTIS 
Technical  Director 
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BATTLE  STAFF  TRAINING  SYSTEM  H:  COMPUTER-BASED  INSTRUCTION 
SUPPORTING  THE  FORCE  XXI  TRAINING  PROGRAM 

EXECUTIVE  SUMMARY 


Research  Requirement: 

Army  commanders  and  staffs  need  structured  training  support  packages  (TSPs)  for 
training  and  assessment  of  individual  combat  skills.  The  existing  Battle  Staff  Training  System 
(BSTS),  a  combination  of  text-based  and  computer-based  lessons,  provides  28  TSPs  for  this 
purpose.  The  advent  and  expansion  of  computer-based  training  (CBT)  will  result  in  less  text- 
based  training  material.  In  addition  to  developing  expanded  CBT  TSPs  for  commanders  and 
staffs,  training  developers  and  CBT  authors  must  know  how  to  maintain  the  currency  of  CBT 
TSPs.  The  purpose  of  this  research  effort  was  to  convert  one  of  the  existing  BSTS  courses  to 
100%  CBT  (excluding  job  aids)  as  well  as  to  design  and  develop  CBT  courses  to  prepare  training 
developers  and  CBT  authors  to  update  and  maintain  the  existing  TSPs. 

( 

Procedure: 

Innovative  Tools  and  Techniques  for  Brigade  and  Below  Staff  Training  II  -  Battle  Staff 
Training  System  II  (ITTBBST-BSTS  II)  was  developed  through  the  use  of  the  Systems  Approach 
to  Training  (SAT)  process  as  applied  to  the  design  and  development  of  interactive  multimedia 
courseware. 

The  target  audience  for  each  course  was  analyzed  in  light  of  the  tasks  required  of  the 
incumbent  in  the  performance  of  his  or  her  duties.  The  analysis  for  converting  the  Brigade 
Common  Core  course  included  a  review  of  previous  BSTS  job  analyses,  review  of  applicable 
doctrinal  publications,  review  of  lessons  learned  available  from  published  material  or  military 
subject  matter  experts  (SMEs),  and  review  and  approval  by  the  customer.  The  review  for  the 
Training  Developer  and  CBT  Author  courses  required  a  bottoms-up  approach  to  identify  the 
required  performance  measures  since  these  were  new  courses  and  were  not  based  on  any 
established  military  procedures.  This  analysis  was  conducted  by  the  contractor  experts  who 
produced  the  BSTS  courseware. 

The  design  process  used  the  results  of  the  job  analyses  to  determine  tasks,  identify  which 
of  these  tasks  were  critical,  produce  instructional  objectives,  develop  criteria  test  items,  prepare 
course  maps,  develop  course  strategies,  and  prepare  examinations.  Formative  evaluation  was 
initiated  early  in  the  design  phase,  and  continued  throughout  development.  Development  of 
course  material  was  based  on  the  analysis  and  design  completed  by  the  research  team.  The 
prototype  courses  were  implemented  on  a  trial  basis  through  SME  and  target  audience  testing  at 
Fort  Knox,  Kentucky,  and  Fort  Benning,  Georgia.  Results  of  testing  were  used  to  revise  each 
TSP. 
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Findings: 


While  the  sample  size  was  relatively  small,  results  from  this  research  effort  provide 
positive  support  for  CBT.  The  time  to  complete  the  Brigade  Common  Core  course  was  reduced, 
on  the  average,  by  40%  and  the  end  of  course  comprehensive  exam  scores  increased.  These 
results  are  consistent  with  other  research  projects  on  the  benefit  of  CBT.  Since  the  Training 
Developer  and  CBT  Author  courses  were  new,  there  is  no  before  and  after  comparison  available. 
However,  participant  feedback  and  performance  results  indicate  that  these  courses  will  be 
extremely  effective  in  preparing  U.S.  Army  Training  and  Doctrine  Command  (TRADOC)  school 
personnel  to  review  and  modify  the  library  of  BSTS  courseware. 

Utilization  of  Findings: 

The  100%  Brigade  Common  Core  course  developed  in  ITTBBST-BSTS II  has  shown 
itself  to  be  effective  in  improving  staff  officer  skills  and  knowledge  in  individual  tasks. 
Previously,  training  developers  and  CBT  authors  had  no  materials  available  to  train  them  in  how 
to  update  and  maintain  existing  BSTS  courseware.  The  Training  Developer  and  CBT  Author 
courses  developed  in  this  project  are  critical  to  filling  this  void.  Also,  the  CBT  method  of 
instruction  received  high  value  ratings  from  target  audience  participants  and  would  appear  to 
lend  itself  to  further  development  and  expansion  as  a  method  of  individual  training.  These 
products  add  to  the  library  of  individual  training  tools  available  to  support  combat  units  and 
TRADOC  schools.  The  training  developer/author  courses  enable  TRADOC  to  update  and 
expand  the  library.  The  lessons  learned  will  help  future  development  teams  design  and  develop 
new  CBT  courses  for  individual  training. 
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BATTLE  STAFF  TRAINING  SYSTEM  H:  COMPUTER-BASED  INSTRUCTION 
SUPPORTING  THE  FORCE  XXI  TRAINING  PROGRAM 


Introduction 

In  support  of  the  Force  XXI  Training  Program  (FXXITP),  the  U.S.  Army  Research 
Institute  for  the  Behavioral  and  Social  Sciences  (ARI)  has  conducted  research  on  various  training 
strategies  for  preparing  individuals  and  staff  groups  at  battalion  and  brigade  level  to  perform  their 
combat  tasks.  The  Infantry  Forces  Research  Unit  (EFRU)  at  Fort  Benning,  Georgia,  began  with 
an  effort  to  identify  critical  combat  skills  required  of  commanders  and  staffs  based  on 
weaknesses  in  unit  performance  at  the  Combat  Training  Centers.  This  evolved  into  the 
development  of  selected  text  and  computer-based  training  (CBT)  training  support  packages 
(TSPs)  for  staff  officers  at  battalion  and  brigade  level  (Andr6  &  Salter,  1996a,  1996b).  The 
Armored  Forces  Research  Unit  (AFRU)  at  Fort  Knox,  Kentucky,  expanded  this  effort  by 
developing  CBT  courses  for  commanders,  revising  the  existing  staff  officer  TSPs  to  include 
updated  doctrine,  and  adding  comprehensive  end  of  course  evaluation  components  (COMPS)  to 
all  courses  based  on  standard  scenarios  used  in  other  training  programs  (Andrd,  Wampler  & 
Olney,  1997).  Also,  in  two  related  projects,  the  IFRU  pursued  the  development  of  100%  CBT 
courses  to  fill  current  training  requirements  (Wampler,  Centric  &  Salter,  1998a,  1998b). 

Throughout  the  above  efforts,  the  AFRU  has  worked  closely  with  the  FXXITP,  an 
initiative  designed  to  carry  the  Army's  training  capabilities  into  the  21st  Century  (U.S.  Army 
Training  and  Doctrine  Command  [TRADOC],  1994).  The  effort  began  with  several  research 
projects.  One  of  those  was  the  Innovative  Tools  and  Techniques  for  Brigade  and  Below  Staff 
Training  (TTTBBST)  program.  The  ITTBBST  program  consisted  of  three  interrelated  efforts. 

One  component  was  analyzing  Battlefield  Functions  which  established  the  common  basis  for 
designing  individual  and  staff  training  programs  (Ford,  Mullen  &  Keesling,  1997).  Next  was  the 
development  of  the  Battle  Staff  Training  System  (BSTS)  courses  to  train  individual  commanders 
and  staff  members  on  planning  and  preparing  for  combat  (Andre  et  al.,  1997).  The  final 
component  of  this  program  developed  the  Staff  Group  Trainer,  an  innovative  tool  for  training 
small  groups  of  staff  officers.  This  component  provides  a  training  link  between  the  individual 
staff  training  courses  in  BSTS  and  collective  training  exercises  involving  the  integrated  staff 
(Koger  et  al.,  1998). 

The  ITTBBST-BSTS  U  program  is  built  on  the  lessons  learned  from  ITTBBST-BSTS 
(Andre  et  al.,  1997).  Previous  research  indicated  that  the  BSTS  courseware  improved  staff 
officer  skills  and  performance  as  evidenced  by  an  average  of  more  than  20%  improvement 
between  pretest  and  posttest  results.  Also,  test  participant  results  indicated  that  CBT  was  a 
preferred  form  of  instruction  over  reading  text-based  lessons  and  doctrinal  reference  manuals. 
Additionally,  there  was  a  recognized  requirement  to  update  and  maintain  the  BSTS  instructional 
material  based  on  evolving  doctrine.  Individuals  would  need  to  be  trained  to  review  and  modify 
the  existing  text  and  computer-based  courseware  to  incorporate  necessary  changes. 
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Background 


As  discussed  by  Andrd  et  al.  (1997),  the  U.S.  Army  is  exploring  and  implementing 
various  distance  learning  techniques  to  meet  the  challenge  of  complex,  highly  diverse  training 
requirements  with  increasingly  constrained  resources.  Given  the  tenets  of  the  Army  Distance 
Learning  Plan  (Department  of  the  Army  [DA],  1996),  TRADOC  has  set  a  course  to  implement 
distance  learning  as  the  preferred  method  of  instruction  now  and  in  the  future.  Training  research 
conducted  by  both  the  AFRU  and  IFRU  of  ARI  has  resulted  in  a  library  of  text  and  computer- 
based  TSPs  that  can  be  used  for  distance  learning  at  schools,  in  units,  or  in  a  stand-alone  mode 
by  staff  officers. 

Unfortunately,  full  implementation  of  a  distance  learning  train-the-trainer  program  for 
government  training  developers  and  CBT  authors  will  not  occur,  based  on  the  current  schedule, 
until  early  2000  (DA,  1996).  Government  training  developers  must  develop  their  skills  from  on- 
the-job  training  and  may  not  be  knowledgeable  of  how  to  review  and  update  CBT  courses.  In 
addition,  CBT  authors  will  need  to  attend  commercial  courses  of  instruction  to  learn  appropriate 
programming  skills,  but  still  will  not  have  the  necessary  information  to  update  and  maintain 
existing  CBT  courseware.  Training  for  these  CBT  authors  requires  associated  course  registration 
fees  and  travel  costs. 

Results  from  previous  research  with  target  audience  staff  officers  indicate  that  CBT  is  a 
preferred  means  of  presenting  training  (Andre  et  al.,  1997).  As  indicated  above,  the  BSTS 
products  developed  for  the  FXXITP  are  a  combination  of  text  and  CBT  materials.  With  an 
expanding  availability  of  computers  for  staff  officers  and  increased  access  to  the  Internet,  an 
alternative  to  printed  instructional  material  could  be  beneficial.  One  possibility  is  100%  CBT 
courseware. 

Fletcher  (1995)  summarized  research  efforts  addressing  CBT  applications  in  military 
training.  He  found  reports  of  reductions  in  training  time  as  well  as  positive  student  attitudes  for 
computer  managed  instruction.  Still  other  reports  address  better  test  performance  from  using 
computerized  training  aids  as  contrasted  with  text-based  workbooks.  Fletcher  also  found  reports 
of  reduced  training  costs  (Fletcher,  1995).  More  details  contained  in  Fletcher’s  summary  are 
available  in  Andre  et  al.  (1997). 

Olmstead  (1992)  stated  that: 

Evidence  has  been  mounting  to  support  the  concept  that  maximum  effectiveness  can  be 
achieved  only  when  a  battle  staff  directly  addresses  the  quality  of  its  organizational 
functioning  and  develops  capabilities  that  enable  it  to  maintain  functional  integrity  under 
the  stress  of  battlefield  pressures,  (p.  11-10) 


He  continues,  stating  successful  performance  of  these  battle  staff  functions  is  dependent,  in  part 
at  least,  on  two  skill  sets.  Battle  staff  members  must  be  proficient  in  their  individual  role  specific 
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skills,  as  well  as  the  team  performance  skills  that  are  needed  for  the  collective  execution  of  team 
processes  (Olmstead,  1992).  The  BSTS II  program  builds  on  Olmstead’s  research  by  developing 
a  100%  CBT  individual  battle  staff  training  course  to  enhance  the  performance  of  organizational 
processes  by  providing  training  for  the  individual,  thereby  enhancing  the  performance  of  the 
organization. 

Peters  (1997)  states: 

Government  agencies  are  increasingly  turning  to  distance  learning  as  an  affordable  means 

of  keeping  employees  trained  and  informed  in  the  ever-changing  federal  workforce.... 

Affordable,  flexible  training  for  large  numbers  of  workers  has  become  critical  as  agencies 

cope  with  tremendous  organizational  and  technological  changes  brought  on  by 

downsizing  and  implementation  of  governmental  reforms,  (p.  31) 

The  increasing  development  of  CBT  courses  mandates  that  government  training  developers  and 
CBT  authors  be  trained  in  updating  and  maintaining  this  newly  evolving  courseware. 

Bloom  (1984)  suggests  that  students  taught  one-on-one  by  an  individual  instructor  can 
demonstrate  greater  achievements  than  those  taught  in  a  classroom  group  of  30.  The  North 
Atlantic  Treaty  Organization  Research  Study  Group  16  states  that  instructional  efficiency  can  be 
gained  when  the  training  is  tailored  to  the  needs  of  the  student.  This  efficiency  is  becoming  a 
necessity  as  military  training  budgets  diminish.  However,  providing  an  instructor  to  each  student 
is  not  economically  feasible.  The  study  group  concluded  that  CBT  provided  an  economic 
solution  by  substituting  the  cost  of  CBT  development  for  the  requirement  to  provide  human 
instructors  (Fletcher,  1995). 

The  value  of  CBT  to  military  trainers  is  significant.  The  BSTS  II  program  applies  the 
outcome  of  references  cited  above  in  providing  three  TSPs  that  can  enable  one-on-one  instruction 
in  a  resource  effective  manner  for  both  military  and  civilian  training.  In  addition,  since  CBT  is 
essentially  a  self-paced  and  independent  study  vehicle,  it  reduces  the  scheduling/conflict 
resolution  requirements  inherent  in  classroom  and  learning  center  training.  Also,  costs  and  time 
lost  in  instructor  and  student  travel  are  reduced  or  negated,  depending  on  the  method  of  CBT 
employed. 

Statement  of  the  Problem 

There  is  currently  no  training  program  or  course  that  instructs  the  government  training 
developer  how  to  review  and  update  the  BSTS  courseware.  A  training  program  is  necessary  to 
provide  the  capability  to  modify  the  courses  as  doctrine  changes.  Also,  previous  research 
indicates  that  CBT  reduces  the  required  training  time  for  students.  While  the  library  of  staff 
officer  training  courses  in  BSTS  is  valuable,  with  the  limited  training  time  available  to  staff 
officers,  courses  could  possibly  be  more  beneficial  if  the  instructional  materials  were  produced  as 
100%  CBT.  Before  expending  limited  resources  to  convert  all  BSTS  courseware,  research 
should  be  conducted  to  determine  the  value  of  100%  CBT  courseware. 
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Previous  research  conducted  by  ARI  (Andrd  et  al.,  1997)  served  as  the  foundation  for  this 
program.  Lessons  learned  from  that  program  were  the  starting  point  for  this  effort.  In  addition, 
two  revised  doctrinal  publications  were  used  in  updating  the  Brigade  Common  Core  course: 

Field  Manual  (FM)  101-5,  Staff  Organization  and  Operations  (DA,  1997b),  and  FM  101-5-1, 
Operational  Terms  and  Symbols  (DA,  1997a). 

The  BSTS II  program  consisted  of  three  TSPs  that,  based  on  previous  lessons  learned, 
were  developed  as  100%  CBT.  All  instructional  material  is  contained  in  a  computer-based 
format.  There  is  limited  adjunctive  text-based  material. 

One  training  package  was  a  conversion  of  the  existing  text  and  computer-based  Brigade 
Common  Core  TSP  into  a  100%  CBT  course.  The  course  was  designed  to  train  common 
individual  staff  officer  skills  required  to  plan  and  prepare  for  combat.  This  conversion  also 
included  updates  to  bring  the  course  in  line  with  current  doctrine.  This  revised  course  was 
developed  as  a  potential  replacement  for  the  existing  BSTS  TSP.  It  is  for  use  by  the  total  force, 
Active  and  Reserve,  and  can  be  used  in  a  local  area  network  (LAN),  wide  area  network,  or  stand¬ 
alone  computer  mode. 

The  second  training  package,  the  Training  Developer  course,  was  designed  to  train  a 
government  subject  matter  expert  (SME)  how  to  review  existing  BSTS  courseware,  including 
both  text  and  CBT  material.  The  Training  Developer  course  explains  how  to  update  and  revise 
the  text-based  lessons  and  provides  a  process  to  convey  needed  computer-based  material  changes 
to  a  CBT  author.  This  course  was  designed  for  use  in  a  TRADOC  school  or  similar  institution 
charged  with  responsibility  for  updating  and  maintaining  BSTS  courseware.  Like  the  existing 
BSTS  TSPs,  the  Training  Developer  course  can  operate  in  any  computer  mode. 

The  third  TSP,  the  CBT  Author  course,  trains  CBT  authors  in  the  skills  and  knowledge 
required  to  incorporate  changes  into  the  computer-based  portions  of  the  TSPs  developed  under 
the  ITTBBST-BSTS  research  program,  based  on  the  revisions  identified  by  the  training 
developer.  Like  the  Training  Developer  course,  the  CBT  Author  course  was  designed  for  use  in 
a  TRADOC  school  or  similar  institution  charged  with  responsibility  for  updating  and 
maintaining  BSTS  courseware.  The  course  was  designed  only  to  operate  in  a  stand-alone 
computer  mode. 

Technical  Objectives 

The  BSTS  II  program  was  designed  to  address  the  following  technical  objectives: 

1.  To  convert  an  existing  BSTS  course  to  100%  CBT  and  formatively  evaluate  the 
prototype  course. 

2.  To  design,  develop  and  formatively  evaluate  two  new  100%  CBT  courses;  one  for 
government  training  developers  and  one  for  CBT  authors.  These  courses  were  designed  to  train 
personnel  how  to  update  and  maintain  BSTS  courseware. 
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Method 


Project  Overview 

The  two  technical  objectives  established  the  requirements  for  this  project.  First,  there 
was  a  need  to  establish  a  100%  CBT  course  for  staff  officers  and  test  its  acceptability.  Second, 
there  was  a  requirement  to  develop  a  means  to  train  government  training  developers  and  CBT 
authors  to  update  and  maintain  the  existing  BSTS  library.  The  BSTS II  project,  using  the 
Systems  Approach  to  Training  (SAT),  applied  the  same  process  as  was  used  in  BSTS  (Andrd  et 
al.,  1997). 

Analyze 

The  initial  step  was  to  determine  the  specific  tasks  to  be  trained  in  each  of  the  courses.  A 
front  end  analysis  was  conducted  along  with  a  determination  of  the  intended  target  population. 
The  other  key  part  of  this  step  was  to  identify  the  appropriate  references  to  be  used  in  developing 
the  course  content. 

Since  the  Brigade  Common  Core  course  already  existed,  this  was  a  relatively  straight 
forward  process.  A  task  analysis  determined  that  no  changes  were  needed  to  the  tasks  to  be 
trained  and  the  target  audience  remained  unchanged.  A  doctrinal  publication  review  found  that 
two  publications  had  been  updated  since  the  BSTS  course  was  developed.  These  manuals  (FM 
101-5  and  FM  101-5-1)  served  as  the  foundation  for  about  half  of  the  course  material.  The 
military  decision-making  process  (MDMP)  documented  in  FM  101-5  (DA,  1997b)  was  changed 
from  a  situational  dependent,  three-model  concept,  to  a  single  model  for  all  situations.  An  entire 
subject  on  the  MDMP  in  the  existing  BSTS  course  was  outdated  based  on  the  latest  FM  101-5. 

In  addition,  a  revised  FM  101-5-1  (DA,  1997a)  presented  several  changes  to  doctrinal  terms, 
definitions,  and  graphic  symbols.  This  meant  a  lesson  on  military  graphics  and  symbols  was 
outdated.  Additionally,  military  graphics  and  terminology  within  all  of  the  course  material  had  to 
be  reviewed  to  ascertain  if  changes  were  required.  To  ensure  that  the  doctrinal  material  in  the 
Brigade  Common  Core  course  was  doctrinally  correct,  the  latest  doctrinal  material  was  included, 
despite  the  fact  that  this  course  would  now  be  inconsistent  with  other  BSTS  courses. 

Since  the  Training  Developer  and  CBT  Author  courses  were  new,  the  target  audience 
needed  to  be  defined.  The  ARI  personnel  assisted  in  identifying  the  anticipated  skills  of  the 
military  and  civilian  personnel  who  would  be  tasked  to  update  and  maintain  the  BSTS  courses. 
The  only  existing  reference  for  reviewing  and  updating  the  BSTS  courseware  were  the 
procedures  developed  and  used  by  the  contractor.  A  review  of  these  procedures  along  with 
interviews  of  training  developers  and  CBT  authors  who  were  involved  in  the  development  of 
BSTS  resulted  in  identification  of  the  critical  tasks  to  be  included  in  the  courses. 

The  resulting  job  analysis  led  to  the  conclusion  that  the  Training  Developer  course  should 
train  the  student  not  only  to  review  the  text  and  CBT  material,  but  also  to  make  necessary 
changes  to  text-based  lessons.  The  results  from  this  phase  of  the  SAT  process  led  to  the  next 
step,  design. 
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Design 


Park  and  Hannafin  (1993)  found  that  the  development  of  interactive  multimedia  is  based 
more  on  technological  capacity  than  research  and  theory.  They  concluded  that  “the  guidelines  for 
multimedia  design  are  based  not  on  empirical  evidence,  but  on  the  intuitive  beliefs  of  designers” 
(1993,  p.  63).  They  proposed  a  framework  for  organizing  research  and  theory  related  to 
interactive  multimedia  design  that  includes  psychological,  pedagogical,  and  technological 
foundations.  The  design  of  BSTS II  employs  some  guidelines  suggested  by  Park  and  Hannafin, 
namely  the  concepts  of  prior  knowledge,  symbol  systems,  feedback  in  CBT,  instructional  control 
in  CBT,  and  the  capability  of  interactive  multimedia. 

Additional  research  suggests  leaving  the  pace  of  the  lesson  under  the  control  of  the  user, 
avoiding  the  placement  of  too  much  text  on  a  screen  display,  and  providing  the  learner  with 
instructions  on  what  to  do  next  (Gagnd,  Wagner  &  Rojas,  1981).  These  suggestions  have  been 
incorporated  into  the  BSTS  II  TSPs.  The  BSTS  13  research  applied  the  instructional  prescriptions 
for  learner  control  as  proposed  by  Chung  and  Reigeluth  (1992): 

There  has  been  a  great  deal  of  research  in  the  area  of  learner  control,  which  is  widely 
believed  to  be  a  highly  desirable  feature  of  individualized  and  interactive  learning 
systems  because,  it  is  said,  this  enables  learning  to  be  individualized  to  each  person’s 
needs.  Unfortunately,  research  findings  regarding  the  effects  of  learner  control  have  been 
inconclusive,  and,  what  is  more,  they  have  been  more  frequently  negative  than  positive. 

(p.  14) 

The  researchers  propose  an  integration  of  learner  control  methods  (content  control,  sequence 
control,  pace  control,  display  control,  internal  processing  control,  and  advisement  strategy)  into  a 
form  useful  to  the  design  of  instruction.  The  TSPs  developed  in  BSTS  II  applied  content  control, 
sequence  control,  and  pace  control. 

The  design  process  used  for  BSTS  II  was  identical  to  the  process  for  BSTS  (Andrd  et  al., 
1997).  The  critical  task  lists  and  draft  course  maps  for  each  course  were  reviewed  by  ARI  and 
members  of  the  FXXITP  team  prior  to  course  development.  Tutorials,  practical  exercises,  media 
selection,  and  assessment  components  (tests)  were  designed  into  the  courses  following  the 
criteria  used  in  designing  the  original  BSTS  courses. 

There  were  two  major  differences  in  the  testing  design.  First,  a  COMPS  component  was 
not  included  for  the  Training  Developer  and  CBT  Author  courses.  The  BSTS  COMPS  were 
based  on  operational  scenarios  and  were  intended  to  have  staff  officers  apply  newly  learned  skills 
to  combat  situations.  This  requirement  was  not  appropriate  for  the  training  developer/CBT 
author.  Second,  there  were  no  subject  pretests  for  the  CBT  Author  course.  It  was  determined 
that  students  completing  this  course  would  not  be  familiar  with  the  design  structure  for  the  BSTS 
courseware  and,  therefore,  there  was  no  need  to  assess  prior  knowledge. 
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Even  though  the  design  process  was  the  same,  there  is  one  significant  point  worth 
discussing  in  this  area.  The  designs  for  these  three  courses  originally  indicated  that  all  material 
would  be  produced  in  CBT;  there  would  be  no  text-based  material.  The  plan  was  to  include 
navigation  and  operating  instructions  at  the  beginning  of  each  course  in  CBT  format.  Following 
the  initial  design  of  the  courses,  ARI  determined  that  certain  adjunctive  material  would  be  useful 
in  a  text-based  workbook  for  student  reference.  While  all  instructional  material  for  each  course 
was  to  be  CBT,  a  student  workbook  would  include:  an  introduction  to  the  course  with  operating 
and  navigation  instructions,  a  course  map,  job  aids,  and  a  reference  list  for  various  help  sources. 

Formative  evaluation  (FE)  was  initiated  early  in  the  design  phase,  and  continued 
throughout  the  development  of  the  courses.  The  FE  procedures  are  discussed  later  in  this  report. 

Develop 

While  the  BSTS  II  development  process  was  similar  to  the  BSTS  process,  there  were  a 
number  of  differences.  These  differences  resulted  from  two  main  factors.  Lessons  learned  from 
other  CBT  development  projects  identified  some  improved  techniques  that  were  implemented. 
Also,  the  large  volume  of  CBT  in  each  course  as  well  as  the  requirement  to  have  multiple 
software  programs  operating  simultaneously  during  a  course  caused  a  variety  of  programming 
changes. 

Authoring  into  CBT  for  the  Brigade  Common  Core  and  Training  Developer  courses  was 
accomplished  using  IconAuthor™  software,  assuring  compatibility  with  all  BSTS  program 
courseware.  The  courses  were  also  developed  to  operate  within  the  training  management  system 
(TMS)  used  for  the  BSTS  courseware. 

Programming  the  Brigade  Common  Core  course  presented  a  challenge  in  linking  two 
compact  discs  (CDs)  which  contained  instructional  material  for  the  same  subject  (Subject  4,  The 
Military  Decision-Making  Process).  This  problem  was  not  encountered  in  previous  BSTS 
courseware.  Risks  were  analyzed,  and  it  was  determined  that  many  files  would  need  to  be 
duplicated  on  both  CDs.  These  files  essentially  were  all  TMS  linked  files  -  subject  menu  files,  as 
well  as  subject  and  lesson  tests.  The  objective  was  to  make  all  transitions  transparent  to  the 
student.  This  meant  that  the  student  would  only  be  required  to  change  CDs  at  an  on-screen 
prompt.  Also,  he  or  she  could  move  back  and  forth  between  CDs.  Finally,  the  changed  CD 
methodology  would  work  within  the  TMS  to  preclude  any  fatal  errors  while  ensuring  the  student 
received  credit  for  reviewing  the  courseware  and  any  tests  taken. 

The  solution  to  this  problem  required  that  menu  items  be  repeated  on  both  CDs.  The 
message  to  change  CDs  was  placed  between  the  menu  selection  and  the  actual  structure  that 
called  the  lesson.  Once  the  message  was  acknowledged  by  the  student  (who  was  required  to 
replace  the  CD,  let  it  “spin  up,”  and  then  click  “OK”)  the  program,  which  is  linear,  proceeded  to 
call  the  next  lesson.  Both  CDs  were  structured  in  this  manner,  which  allowed  free  movement 
between  CDs,  in  either  direction. 
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Developing  the  Training  Developer  course  incorporated  techniques  to  improve  skill 
retention  and  allow  the  quickest  possible  access  to  major  topic  areas  of  instruction.  For  example, 
even  though  the  student  is  not  familiar  with  BSTS  courseware  development,  a  pretest  is  provided 
at  the  start  of  each  subject.  This  alerts  the  student  to  the  type  of  material  to  be  presented  in  the 
course.  Quizzes  were  included  following  most  major  topics  to  help  reinforce  the  instructional 
material.  Numerous  samples  for  developing  text-based  material  were  taken  from  the  course 
development  procedures  and  included  in  the  CBT  as  well  as  the  student  workbook.  A  variety  of 
“screen  captures”  from  various  program  software  packages  and  BSTS  courses  were  built  into  the 
courseware  so  the  student  could  experience  actually  reviewing  CBT  material.  A  variety  of 
review  methods  were  presented  so  the  student  had  flexibility  in  reviewing  CBT  material, 
depending  on  the  type  and  amount  of  material  that  needed  to  be  reviewed.  Finally,  the 
courseware  was  structured  in  a  modular  fashion  so  a  person  desiring  to  review  a  single  training 
developer  skill  could  quickly  and  easily  access  the  desired  topic. 

The  CBT  Author  course  was  developed  for  personnel  who  had  attended  a  minimum  of 
the  IconAuthor™  Basic  course,  and  preferably  the  advanced  course.  This  would  preclude  having 
to  teach  IconAuthor™,  and  allow  the  course  to  focus  on  BSTS-specific  programming 
conventions.  This  was  critical  to  ensure  that  any  future  modifications  to  BSTS  courses  did  not 
destroy  links  to  the  TMS  and  that  course  conventions  (checkmarks,  bookmarks,  testing,  and 
menus)  were  understood.  The  course  was  intended  to  instruct  the  student  on  what  information 
was  contained  in  which  files,  and  how  to  quickly  locate  page  or  structure  problems.  The  problem 
initially  was  to  determine  the  best  method  of  delivery. 

The  desired  design  of  the  CBT  Author  course  was  to  “coach”  the  student  through  some  of 
the  more  difficult  programming  concepts,  such  as  navigational  loops  and  branching,  while 
actually  in  the  authoring  program.  This  required  the  computer-based  lesson  to  run  at  the  same 
time  the  authoring  program  was  open.  However,  IconAuthor™  does  not  allow  the  runtime  player 
(Present)  and  the  authoring  software  to  run  simultaneously.  It  was  necessary  to  find  a  program 
that  would  allow  the  IconAuthor™  authoring  program  to  run  while  the  presentation  (coach) 
program  could  operate  in  its  own  runtime  player.  Initial  experiments  by  the  CBT  team  indicated 
that  ToolBook  H™  would  run  while  IconAuthor™  was  open.  In  some  of  the  more  difficult 
lessons  a  ToolBook  II™  sidebar  as  shown  in  Figure  1  was  added  to  the  screen  that  ran  a  menu 
and  a  means  to  repeat  audio.  Next  to  the  sidebar  ran  IconAuthor™  in  the  authoring  mode.  The 
lesson  literally  coached  the  student  through  activities,  using  the  IconAuthor™  screen  mapping 
system  to  locate  structure  and  icons. 
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Figure  1.  A  representation  of  a  ToolBook  H™  sidebar  with  IconAuthor™  in  the  authoring  mode. 

Another  difference  in  the  CBT  Author  course  involved  the  use  of  Lotus  ScreenCam™  to 
show  the  student  how  something  was  done,  in  many  cases  including  narration.  This  would  make 
it  easier  for  the  student  to  understand,  and  it  would  allow  the  use  of  the  “teach/show/do”  method 
of  training.  ScreenCams™  were  to  be  used  in  two  methods:  one  where  simple  tasks  are  shown, 
and  another  where  students  are  shown  how  to  do  a  complex  task  and  then  are  required  to  perform 
the  same  task.  Students  could  call  on  an  audio  coach  to  assist  them.  A  number  of  problems  were 
encountered  while  using  ScreenCam™. 

The  person  recording  the  ScreenCam™  needed  to  be  the  SME  for  the  material,  not  an 
audio/video  specialist.  It  was  too  difficult  and  cost  prohibitive  to  attempt  this  with  the  SME 
explaining  to  the  audio/visual  person  what  to  do  and  how  to  do  it,  and  still  ensure  that  the 
narration  and  video  were  synchronized.  Even  though  someone  produced  the  narration  other  than 
a  professional  narrator,  the  result  was  well  received  by  beta  test  personnel,  who  felt  that  the 
instruction  was  informal,  friendly,  and  easy  to  understand. 

The  second  problem  encountered  was  the  speed  of  playback  and  the  synchronization  of 
audio  with  the  ScreenCam™.  While  both  are  recorded  simultaneously  through  ScreenCam™, 
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playback  synchronization  would  sometimes  be  lost  and  either  audio  or  video  would  get  ahead  of 
the  other.  Playback  speed  also  seemed  to  be  a  problem,  in  that  it  appeared  faster  than  what  was 
originally  recorded.  This  caused  recordings  to  be  done  much  slower  and  with  extreme 
deliberation  to  create  the  proper  effect  on  playback. 

Another  problem  was  an  error  message  created  by  ScreenCam™  that  stated  it  could  not  be 
played.  However,  after  clicking  “OK”  in  the  error  message  box,  the  ScreenCam™  played 
normally.  Information  obtained  from  Lotus  indicated  that  the  current  version  of  ScreenCam™ 
sometimes  has  conflicts  with  Windows  95  drivers,  causing  the  message  to  appear.  Clicking 
“OK”  would  allow  ScreenCam™  to  play  normally.  The  problem  was  a  software  bug  that  would 
be  fixed  in  future  versions.  Since  the  value  of  being  able  to  show  a  student  how  to  do  something 
outweighed  the  annoyance  of  the  error  message,  it  was  decided  to  keep  the  ScreenCam™  shots 
and  instruct  the  students  what  to  do  when  or  if  they  encountered  the  error  message. 

The  next  problem  was  the  selection  of  the  file  format  to  use  with  ScreenCam™.  There 
were  three  choices,  AVI  Video  for  Windows  format,  SCN  ScreenCam™  file  format,  and 
conversion  to  an  executable  file.  We  quickly  found  that  AVI  file  formats  were  extremely  large 
(many  as  large  as  30  megabytes)  so  this  option  was  discarded.  Use  of  SCN  ScreenCam™  files 
requires  the  use  of  the  Lotus  ScreenCam™  player.  This  would  require  the  student  to  install  one 
more  program  to  his  computer,  so  this  choice  was  eliminated.  Converting  the  ScreenCam™  files 
to  an  executable  format  proved  to  be  the  best  choice.  We  then  discovered  that  ToolBook  D™ 
required  that  executable  files  be  placed  in  the  same  directory  every  time.  In  order  to  make  the 
solution  seamless  to  the  student,  the  install  set  that  copied  the  ToolBook  II™  runtime  player  to 
the  student’s  computer  also  copied  the  ScreenCam™  files  to  the  same  drive.  When  the  student 
uninstalled  the  program,  the  installation  set  would  also  remove  the  ScreenCam™  file  directory. 

The  CBT  Author  course  required  multiple  CDs,  just  as  the  Brigade  Common  Core 
course.  We  initially  tried  the  same  concept  used  for  the  Brigade  Common  Core,  using  dual 
menus  to  drive  the  CD  change  process.  This  did  not  work  due  to  the  fact  that  ToolBook  II™ 
Instructor  operates  using  an  entirely  different  paradigm  from  IconAuthor™.  It  became  necessary 
to  create  two  install  sets,  one  for  each  CD.  The  install  sets  created  a  program  folder  and  menu 
selection  items  from  the  Windows  start  menu  for  CD1  and  CD2.  The  menu  selections  indicated 
to  the  student  which  subjects  were  on  which  CDs,  as  did  the  actual  course  map,  which  was  the 
master  menu  for  the  program. 

Testing  was  also  accomplished  differently  with  the  CBT  Author  course  from  other  BSTS 
courseware.  Due  to  the  fact  that  the  BSTS  TMS  and  ToolBook  II™  are  not  compatible,  scores 
could  not  be  recorded,  but  the  student  was  shown  his  score  at  the  end  of  each  test.  A  “login” 
procedure  was  developed  along  with  a  method  to  record  scores  to  a  floppy  disk  in  ASCII  text 
format  that  allowed  a  mentor  or  supervisor  to  read  student  test  question  responses  and  the  overall 
score  for  each  test.  Instructions  in  “how  to”  read  these  floppy  disks  were  included  in  the  printed 
“How  To”  portion  of  the  course. 
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IconAuthor™  and  ToolBook  H™  are  commercially  available  software  programs; 
therefore,  the  source  code  of  BSTS  n  courseware  is  not  retained  by  the  contractor,  but  can  be 
opened  by  anyone  that  possesses  a  copy  of  IconAuthor™  and  ToolBook  D™. 

Implement 

Due  to  the  nature  of  the  BSTS  n  project,  implementation  and  evaluation  were  closely 
interrelated  and  overlapped. 

The  Brigade  Common  Core  and  Training  Developer  courses  were  implemented  on  a  trial 
basis  at  the  Mounted  Battle  Space  Battle  Lab  Test  Bed,  Fort  Knox,  Kentucky.  The  CBT  Author 
course  was  implemented  in  the  developmental  facilities  at  Fort  Benning,  Georgia.  Participants 
(SMEs)  and  target  population  personnel  were  selected,  briefed  and  then  issued  prototype  versions 
of  the  material  a  student  would  receive  when  taking  a  BSTS  II  course.  These  participants 
provided  feedback  and  demographic  data  which  was  returned  to  course  developers  for 
evaluation.  Appendix  B  provides  a  description  of  the  participant  requirements  for  this  program. 

Evaluate 


The  BSTS  II FE  followed  the  same  basic  method  used  in  the  predecessor  BSTS 
program,  with  the  addition  of  a  Phase  0.  Developers  performed  quality  control  checks  on  the 
course  material  throughout  the  design  and  development  process.  Additionally,  test  participants, 
including  doctrinal  SMEs  and  training  developer  experts  as  well  as  target  population  personnel, 
evaluated  the  materials  at  various  phases  of  development.  The  five-phase  process  for  this 
program  is  summarized  below.  See  Andrd  et  al.  (1997)  for  a  more  detailed  explanation  of  the 
process. 

Phase  0  -  Before  any  programming  was  accomplished,  a  full  review  of  draft  storyboards, 
both  paper-based  and  electronic  PowerPoint  slides  was  conducted,  to  validate  content  and 
displays.  This  allowed  the  developer  to  modify  the  storyboards  before  a  CBT  author  began 
programming. 

Phase  I  -  Alpha  testing  by  contractor  personnel  to  ensure  proper  functionality  of  all 
courseware. 

Phase  II  -  Alpha  testing  of  the  complete  (draft)  TSP  by  government  SMEs  at  Fort  Knox. 
This  included  at  least  one  participant  per  course.  Since  no  SME  was  available  for  the  CBT 
Author  course,  this  course  was  tested  at  Fort  Benning  by  a  contractor  expert,  under  the 
supervision  of  two  government  representatives. 

Phase  III  -  Beta  testing  of  the  complete  (draft)  TSP  by  target  population  personnel  at  Fort 
Knox.  This  included  two  to  five  participants  for  each  course.  Most  participants  had  completed 
an  assignment  as  a  battalion  or  brigade  staff  officer,  were  captains  or  Department  of  the  Army 
Civilians,  and  were  branch  qualified  for  schooling.  The  CBT  Author  course  was  again  tested  by 
contractor  personnel  at  Fort  Benning  under  the  supervision  of  government  representatives. 
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Phase  IV  -  Review  of  the  final  (draft)  TSP  by  ARI  and  designated  team  members.  This 
included  at  least  one  participant  for  each  course. 

Procedures.  The  FE  exercises  were  conducted  on  government  provided  hardware 
compatible  with  hardware  and  software  of  previous  BSTS  courseware.  The  100%  CBT  Brigade 
Common  Core  and  Training  Developer  courses  were  integrated  into  the  BSTS  TMS  and  placed 
on  the  LAN  in  the  Mounted  Battle  Space  Battle  Lab  Test  Bed  at  Fort  Knox.  The  CBT  Author 
course  was  not  compatible  with  the  TMS  so  it  was  evaluated  on  a  stand-alone  computer  at  Fort 
Benning.  All  FE  participants  were  coordinated  and  selected  by  the  contracting  officer’s 
representative  (COR).  Analysis  of  FE  results  served  as  the  basis  for  revising  TSPs. 

The  FE  of  the  BSTS  II  courses  included  evaluation  of  all  courseware  (CBT  and  student 
workbook)  and  the  software  providing  the  operating  environment  for  the  computer-based 
courseware.  There  were  a  few  differences  in  FE  procedures  for  BSTS  II  from  the  FE  conducted 
for  the  BSTS  program.  These  differences  are  discussed  below. 

As  outlined  above,  there  were  five  phases  in  this  FE  process.  Phases  I-IV  were  basically 
the  same  as  in  the  BSTS  program  from  the  standpoint  of  objectives  and  implementation 
procedures.  A  new  Phase  0  was  conducted  early  in  the  development  process.  The  COR  and  a 
team  of  government  representatives  reviewed  the  draft  storyboards  for  all  three  courses.  Paper- 
based  storyboards  were  provided  to  the  reviewers  and  comments  were  recorded  directly  on  the 
storyboards.  Also,  electronic  copies  of  the  storyboards  were  available  in  PowerPoint  so  the 
actual  layout  and  color  of  intended  visual  displays  could  be  seen.  A  total  of  six  reviewers  over 
three  days  examined  the  technical  content  of  the  draft  storyboards  and  provided  suggestions  for 
modifying  the  material.  While  this  additional  review  step  required  more  government 
participation,  it  actually  conserved  resources  over  the  course  of  the  program.  Comments, 
changes,  and  corrections  noted  during  this  review  were  incorporated  into  the  courseware  before 
programming  was  accomplished.  Therefore,  participants  in  Phase  II  FE  encountered  fewer 
problems  and  made  considerably  fewer  comments.  This  also  meant  revisions  to  the  courseware 
following  FE  could  be  accomplished  more  quickly. 

Prior  to  the  start  of  each  external  FE  step,  participants  were  briefed  on  the  BSTS  II 
program  as  well  as  the  conduct  of  the  evaluation.  At  least  one  BSTS  II  team  member  was 
present  during  each  external  evaluation  to  observe,  to  record  observations,  and  to  assist 
participants  with  hardware  and  software  issues.  To  measure  the  effectiveness  of  the  courseware, 
FE  data  collection  forms  were  prepared  by  the  BSTS  II  team  and  approved  by  ARI.  Forms  were 
very  similar  to  those  used  in  BSTS  data  collection. 

Having  a  development  team  member  present  during  external  evaluations  was  a  specific 
recommendation  from  the  BSTS  program  which  was  implemented  for  BSTS  n.  There  were  two 
major  benefits  to  this  approach.  First,  there  were  occasions  when  participants  encountered  a 
minor  problem  or  had  a  question  about  the  courseware.  A  knowledgeable  development  team 
member  could  answer  these  questions  and  resolve  most  issues  on  the  spot.  This  saved  time  for 
the  participant  and  also  provided  first  hand  information  to  the  development  team  which 
expedited  the  revision  process.  The  second  benefit  involved  understanding  participant 


12 


comments.  There  were  instances  in  the  BSTS  program  where  the  development  team  members 
had  difficulty  understanding  and  accurately  interpreting  a  participant’s  written  comment  when 
the  FE  forms  were  returned.  During  BSTS  II  the  on-site  development  team  member  could  easily 
clarify  participant  comments,  ensuring  that  problem  areas  were  clearly  understood. 

Government  support  requirements  for  FE  activities  were  minimal  due  to  the  limited 
number  of  TSPs  developed.  Only  14  personnel  from  Fort  Knox  participated  in  Phases  I3-IV.  This 
led  to  the  final  substantive  difference  between  BSTS  and  BSTS  n.  Due  to  the  technical  nature  of 
the  CBT  Author  course,  the  contractor  provided  the  qualified  participant  for  Phase  II  and  HI  FE. 
This  FE  occurred  on  a  stand-alone  computer  at  Fort  Benning.  All  FE  procedures  used  for  the 
other  courses,  from  briefing  the  participant  to  having  the  participant  actually  record  comments  on 
FE  forms,  remained  the  same. 

Revision  of  training  products.  The  results  of  FE  Phases  n,  HI,  and  IV  were  summarized 
and  the  TSPs  were  revised  as  appropriate.  It  is  worth  noting  the  Phase  0  FE  resolved  most  of  the 
technical  content  and  layout  issues  so  there  were  fewer  participant  comments  in  the  later  FE 
phases  (compared  to  FE  results  in  the  BSTS  program). 

Results 

The  output  of  this  program  was  three  TSPs  with  all  instructional  material  contained  in 
CBT.  The  Brigade  Common  Core  course  is  organized  into  four  subjects  that  present  the 
principles  of  Army  operations  doctrine,  the  fundamentals  for  employing  an  armored  or 
mechanized  infantry  brigade,  the  key  staff  functions  that  are  necessary  to  achieve  staff 
integration,  and  the  decision-making  process  as  contained  in  FM  101-5  (DA,  1997b).  All 
instructional  material  is  contained  on  five  CDs.  An  introduction  to  the  course,  a  course  outline, 
and  selected  job  aids  are  contained  in  a  loose-leaf  notebook.  The  end  of  course  COMPS  are 
provided  on  a  separate  CD.  The  necessary  operations  orders  and  overlays  are  also  provided  as 
separate  items.  Based  on  the  results  of  test  participants,  students  required  between  18.5  and  19 
hours  to  complete  the  course,  including  all  tests  and  the  COMPS.  The  students  achieved  an 
overall  90%  average  for  the  lesson  exams. 

All  instructional  material  for  the  Training  Developer  course  is  contained  on  a  single  CD. 
In  addition,  a  loose-leaf  notebook  contains  a  course  introduction,  a  course  outline,  and  numerous 
job  aids  that  are  needed  as  references  to  review  and  update  existing  BSTS  courseware.  The 
course  is  organized  into  two  subjects:  how  to  review  and  make  revisions  to  text-based  lessons  in 
the  BSTS  courses;  and  how  to  review  the  CBT  material  in  BSTS,  then  convey  the  needed 
changes  to  a  CBT  author.  There  are  no  COMPS  for  this  course.  Test  participants  completed  this 
course  in  5  to  5.5  hours,  achieving  an  overall  lesson  exam  score  of  93%. 

The  CBT  Author  course  is  contained  on  two  CDs.  It  also  comes  with  a  loose-leaf 
notebook  with  contents  similar  to  the  Training  Developer  course.  The  course  consists  of  five 
subjects.  Each  subject  addresses  a  different  portion  of  the  CBT  material  in  the  existing  BSTS 
courseware  and  explains  how  to  make  modifications.  Test  participants  completed  the  course  in 
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19  to  19.5  hours,  including  all  tests.  As  with  the  Training  Developer  course,  there  are  no 
COMPS.  The  students  achieved  an  overall  average  of  85%  for  the  lesson  exams. 

Feedback  from  FE  participants  indicated  that  all  courseware  was  easy  to  navigate  and  the 
instructional  material  was  clearly  presented.  They  were  able  to  complete  the  courses  without 
developer  assistance.  Participants  ranked  the  CBT  material,  including  the  tests,  as  having  a 
higher  educational/training  value  than  any  paper-based  materials. 

The  100%  CBT  Brigade  Common  Core  course  trained  the  same  tasks  as  the  text  and 
computer-based  Brigade  Common  Core  course  in  the  existing  BSTS  library.  Even  though  there 
was  no  statistical  control  applied  to  the  group  of  FE  participants  for  these  courses,  there  is  at  least 
one  observation  worth  noting.  The  existing  BSTS  course  required,  on  the  average,  slightly  over 
3 1  hours  for  the  participants  to  complete.  This  included  reading  text-based  lessons  and 
completing  the  CBT  material  (Andr6  et  al.,  1997).  The  revised  course  converted  all  text-based 
lessons  into  CBT  material.  The  average  time  to  complete  this  course  was  less  than  19  hours, 
nearly  a  40%  time  reduction.  A  summary  of  selected  data  from  the  FE  process  is  contained  in 
Appendix  C. 

Lessons  Learned 

The  analysis  phase  of  the  SAT  process  is  absolutely  critical  when  producing  instructional 
material.  Several  key  points  to  emphasize  this  were  encountered  in  this  project.  For  example, 
even  though  the  Brigade  Common  Core  course  already  existed  in  a  combined  text  and  CBT 
format,  it  was  not  sufficient  merely  to  convert  text-based  lessons  into  CBT  lessons.  Two  of  the 
doctrinal  references  had  been  updated  which  caused  about  half  of  the  course  content  to  become 
outdated.  This  required  a  decision  as  to  whether  the  content  of  the  course  should  be  updated 
based  on  the  new  doctrine.  The  course  content  was  updated  to  make  it  current,  but  caution  must 
now  be  used  when  distributing  the  library  because  this  course  is  inconsistent  with  the  other  27 
BSTS  courses.  At  some  point,  the  remaining  BSTS  courses  should  incorporate  the  changes 
based  on  the  new  doctrine. 

Another  key  point  in  the  analysis  phase  is  to  ensure  a  complete  terminal  learning 
objective  (TLO)  for  the  courseware.  In  the  case  of  the  Training  Developer  course,  the  initial 
TLO  stated  that  the  student  should  be  able  to  review  BSTS  courseware  and  identify  needed 
changes.  After  further  consideration,  it  was  determined  that  this  person  should  not  only  identify 
needed  changes  to  all  courseware,  but  should  also  be  responsible  for  actually  executing  changes 
to  text-based  material.  This  required  expanding  the  course  to  include  techniques  and  procedures 
for  editing  the  text  material  as  well  as  samples  of  formats  used  for  the  various  text-based 
materials. 

It  is  imperative  to  involve  the  user  early  in  the  design  phase  of  the  project.  While 
statements  of  work  and  design  plans  attempt  to  accurately  and  completely  articulate  the  expected 
outcome  of  a  project,  ambiguities  and  misinterpretations  will  exist.  By  having  early  and 
continual  dialogue  with  the  user,  variations  in  interpretations  can  be  identified  and  brought  to  a 
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common  understanding.  Obtaining  approval  of  course  designs  early  on  allows  the  project 
resources  to  be  entirely  focused  on  the  expected  outcome. 

Recommendations 


As  with  most  research  projects,  unexpected  contingencies  typically  occur  once  the  actual 
development  begins.  In  the  BSTS  II  project  new  techniques  to  facilitate  student  learning  and 
retention  were  incorporated  into  the  courseware.  These  included  ScreenCams™  to  teach/show/do 
and  having  an  authoring  program  running  to  allow  student  practice  while  the  presentation 
program  is  running.  In  some  cases  this  involved  using  new  computer  software  and  also  using  a 
combination  of  software  that  had  not  been  previously  tried.  It  is  difficult  to  accurately  anticipate 
the  problems  that  might  be  encountered.  Therefore,  research  program  resources  should  allow  for 
these  unexpected  challenges. 

All  required  training  materials  were  contained  in  the  student  workbook  and  the 
accompanying  CDs  with  one  major  exception.  In  the  case  of  the  Brigade  Common  Core  course, 
operation  orders  with  overlays  were  provided,  but  the  actual  map  sheets  were  not  included.  The 
required  maps  may  not  be  readily  available  to  all  students.  Proponents  for  distributing  the 
courseware  should  consider  including  the  needed  map  sheets  in  the  TSP. 

The  inclusion  of  an  FE  phase  to  review  storyboards  early  in  the  development  process 
provided  an  overall  resource  savings  to  this  project.  Identifying  content  and  layout  changes  on 
the  draft  storyboards  allowed  changes  to  be  implemented  before  any  time  and  materials  were 
expended  to  produce  graphics  or  program  the  courseware.  This  FE  technique  was  used  on  other 
projects  (Wampler  et  al.,  1998a,  1998b)  to  develop  CBT,  and  in  every  case  produced  positive 
results.  Future  developers  should  include  this  FE  phase  in  CBT  projects. 

The  adage  that  “a  picture  is  worth  is  thousand  words”  is  certainly  true  when  a  student  is 
completing  a  self-paced  course  of  instruction  without  instructor  interface.  Incorporating 
ScreenCam™  videos  throughout  the  CBT  Author  course  contributed  noticeably  to  learning, 
according  to  FE  feedback.  Videos  should  certainly  be  considered  as  an  instructional  media  in 
CBT  development. 

Depending  on  how  a  CBT  course  is  structured,  it  cannot  only  serve  as  a  training  product, 
but  can  also  be  employed  as  a  job/work  aid.  In  the  case  of  the  CBT  Author  course,  a  person 
involved  in  incorporating  changes  to  BSTS  courseware  could  have  the  CBT  Author  course 
running  on  his  computer  (using  ToolBook  D™)  and  access  selected  topics  for  assistance  in 
performing  BSTS  updates  (using  IconAuthor™).  Future  developers  should  be  mindful  of  this 
multiple  use  possibility  when  designing  CBT  courseware. 
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Discussion 


The  SAT  process  is  a  valuable  tool  when  implemented  properly.  It  is  important  that  each 
of  the  steps  be  reviewed  and  executed.  Circumventing  or  inappropriately  abbreviating  one  of  the 
steps,  especially  analysis,  could  cause  design  and  development  efforts  to  expend  unnecessary 
resources  and  compromise  quality  of  the  finished  product. 

User  involvement  and  FE  are  essential  early  in  the  project  and  throughout  all  steps  of  the 
SAT  process.  Especially  in  research  projects,  many  parameters  are  not  well  defined  at  the  outset 
and  decisions  that  impact  design  and  development  occur  regularly.  With  close  coordination 
between  the  user  and  developer  as  well  as  routine  FE  phases,  informed  decisions  can  be  made  in 
a  timely  manner. 

In  order  to  be  effective,  the  BSTS  library  of  courses  will  require  updating  and 
maintaining.  Doctrine  will  continue  to  change  as  new  equipment  is  fielded  and  evolving 
technologies  alter  the  way  the  Army  fights.  This  will  mandate  changes  to  the  courses.  With  the 
development  of  the  Training  Developer  and  CBT  Author  courses,  the  government  now  has  the 
capability  to  train  personnel  to  perform  these  tasks. 

Based  on  student  comments,  in  both  the  BSTS  II  and  the  BSTS  courses,  students 
preferred  the  CBT  material  rather  than  reading  text-based  material.  While  statistical  comparison 
between  BSTS  and  BSTS  II  test  results  was  not  attempted  in  this  research  effort,  there  is  a  need 
to  evaluate  the  effectiveness  of  100%  CBT  TSPs  in  an  operational  (typical  user)  environment. 

Feedback  from  participants  and  test  results  indicate  that  BSTS  II  courses  are  valuable 
TSPs  and  enhance  the  existing  library  of  TSPs  developed  under  the  BSTS  program.  The 
Training  Developer  and  CBT  Author  courses  should  be  made  available  to  the  agency  responsible 
for  updating  and  maintaining  the  BSTS  courseware. 

The  100%  Brigade  Common  Core  course,  which  includes  the  most  current  doctrine, 
should  replace  the  combined  text  and  CBT  based  course  currently  being  distributed  in  BSTS. 
However,  there  must  be  a  clear  explanation  included  with  the  courseware  that  explains  the 
MDMP  and  military  graphics  instructional  materials  contained  in  this  course  are  based  on  the 
versions  of  FM  101-5  and  FM  101-5-1  published  in  1997,  which  is  inconsistent  with  the  other 
BSTS  courses.  Other  courses  are  still  valuable  since  they  address  the  particular  staff  officer  role 
in  the  MDMP  process,  but  the  fine  points  of  certain  MDMP  steps  are  different.  While  other 
BSTS  courses  are  still  valuable  training  materials,  they  should  be  updated  to  reflect  the  latest 
doctrine  contained  in  evolving  publications,  especially  the  MDMP. 

It  is  clear  from  student  assessments  that  CBT  materials  are  preferred  to  reading  text-based 
lessons.  In  addition,  a  reduction  in  course  completion  time  is  a  distinct  advantage.  Therefore,  a 
future  effort  should  convert  the  existing  BSTS  text-based  lessons  to  CBT. 

As  expected,  courses  developed  with  100%  CBT  require  more  CD  space  to  store  the 
material.  The  ability  to  link  multiple  CDs  during  the  completion  of  courseware  is  a  necessity  and 
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this  process  should  be  transparent  to  the  student.  This  project  used  two  different  linking 
processes  due  to  the  different  operating  software  used  to  program  the  courses.  When  using 
ToolBook  II™,  a  separate  install  set  was  used  for  each  CD.  For  IconAuthor™,  which  uses  a 
different  operating  paradigm,  selected  files  were  copied  onto  both  CDs.  As  additional  CBT 
courseware  is  developed  and  different  operating  systems  are  utilized,  other  means  to  link 
multiple  CDs  may  be  needed.  Other  technologies  and  solutions,  such  as  storing  courses  on  a 
server  or  on  a  large  capacity  hard  drive,  could  alleviate  the  need  for  multiple  CDs. 

The  TRADOC  has  set  a  course  to  implement  distance  learning  as  a  method  of  instruction 
now  and  in  the  future  (DA,  1996).  The  use  of  CBT  is  one  of  the  main  distance  learning 
techniques.  With  the  advent  of  computer  technology  and  the  increased  access  afforded  to  most 
military  personnel,  another  means  of  distributing  the  CBT  courseware  besides  CD  should  be 
pursued.  The  BSTS  courseware  could  be  revised  to  make  it  Internet  delivered  as  opposed  to  CD 
based. 


In  addition  to  revising  the  BSTS  courseware,  the  TMS  should  also  be  adapted  to 
accommodate  newer  technologies.  As  currently  designed,  the  TMS  operates  on  a  stand-alone 
computer  as  well  as  in  a  LAN  or  wide  area  network.  However,  the  network  must  be  a  closed 
domain  for  the  TMS  to  function.  This  limits  access  to  only  those  personnel  within  the  domain. 
An  open  domain  TMS  that  is  Internet  capable  could  increase  the  accessibility  to  the  BSTS 
courses  and  facilitate  the  updating  of  student  results. 

While  preliminary  research  results  indicate  that  CBT  is  an  effective  instructional  media, 
the  value  of  100%  CBT  courses  has  not  been  determined.  Future  research  should  directly  assess 
the  training  effectiveness  and  transfer  of  100%  CBT  material  versus  a  mix  of  CBT  and  text-based 
materials. 

The  Army  Distance  Learning  Plan  (DA,  1996)  is  moving  to  take  the  training  to  the 
student  rather  than  bringing  the  student  to  the  training.  In  this  process,  the  quality  training 
currently  available  at  Army  schools  must  remain  the  standard.  Increasingly,  the  benefits  afforded 
by  CBT  are  making  this  the  solution  of  choice.  The  BSTS  II  program  accomplished  two  major 
objectives  that  enhance  the  Army’s  capability  to  move  more  toward  CBT.  The  development  of 
100%  CBT  courses  breaks  the  link  to  required  text-based  instructional  material  and  the 
government  now  has  an  internal  capability  to  update  and  maintain  the  existing  BSTS  courseware. 
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APPENDIX  A 


Acronyms  and  Abbreviations 
AFRU  Armored  Forces  Research  Unit 

ARI  U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences 

BSTS  Battle  Staff  Training  System 

CAS3  Combined  Arms  and  Services  Staff  School 

CBT  Computer-Based  Training 

CD  Compact  Disc 

COMPS  Comprehensive  End  of  Course  Evaluation  Component 

COR  Contracting  Officer’s  Representative 

DA  Department  of  the  Army 

FE  Formative  Evaluation 

FM  Field  Manual 

FXXITP  Force  XXI  Training  Program 

IFRU  Infantry  Forces  Research  Unit 

ITTBBST  Innovative  Tools  and  Techniques  for  Brigade  and  Below  Staff  Training 

LAN  Local  Area  Network 

MDMP  Military  Decision-Making  Process 

OAC  Officer  Advance  Course 

SAT  Systems  Approach  to  Training 

SME  Subject  Matter  Expert 

TLO  Terminal  Learning  Objective 

TMS  Training  Management  System 

TRADOC  U.S.  Army  Training  and  Doctrine  Command 

TSP  Training  Support  Package 
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APPENDIX  B 


BSTS  D  Formative  Evaluation  Participant  Requirements 

The  following  is  a  description  of  the  personnel  required  to  conduct  FE  of  the  BSTS  II 
courses: 

1.  Brigade  Common  Core  course 

a.  Phase  II  -  Alpha:  Must  be  a  combat  arms  captain  or  major,  who  has  completed 
an  officer  advance  course  (OAC)  and  the  Combined  Arms  and  Services  Staff  School  (CAS3). 
Must  have  served  on  the  staff  of  a  maneuver  brigade  for  at  least  a  year. 

b.  Phase  HI  -  Beta:  Must  be  a  captain  or  major  of  the  appropriate  branch  to  fill  a 
Table  of  Organization  and  Equipment  position  of  a  principal  staff  officer  on  a  maneuver  brigade 
staff.  The  officer  must  have  completed  the  required  schools  and  assignments  commensurate  with 
branch  and  rank  to  be  branch  qualified. 

2.  Training  Developer  course 

a.  Phase  II  -  Alpha:  Must  be  a  combat  arms  captain  or  major,  who  has  completed 
an  OAC,  CAS3,  and  the  TRADOC  training  developers  course  or  training  developers  course  for 
middle  managers.  Must  have  2-3  years  experience  developing  training.  A  civilian  with 
equivalent  experience  is  acceptable. 

b.  Phase  HI  -  Beta:  Must  be  a  combat  arms  lieutenant,  captain,  or  major,  who  has 
completed  an  OAC,  CAS3,  and  the  TRADOC  training  developers  course  or  training  developers 
course  for  middle  managers.  A  civilian  with  equivalent  experience  is  acceptable. 

3.  CBT  Author  course 

a.  Phase  II  -  Alpha:  Must  have  completed  the  basic  and  advanced  IconAuthor™ 
course  and  have  six  (6)  months  programming  experience;  or  have  three  (3)  years  of  programming 
experience  with  IconAuthor™;  or  have  similar  training/experience  in  Microsoft  Visual  C++  or 
Authorware™.  Must  be  proficient  with  Windows  3.xx  and  Windows  95  operating  systems.  Can 
be  either  military  (officer  or  NCO)  or  civilian. 

b.  Phase  HI  -  Beta:  Must  have  completed  the  basic  IconAuthor™  course  and  have 
two  (2)  months  of  programming  experience  within  the  previous  12  months.  Must  be  proficient 
with  Windows  3.xx  and  Windows  95  operating  systems.  Can  be  either  military  (officer  or  NCO) 
or  civilian. 
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APPENDIX  C 


Final  Research  Report  Data 

1.  Throughout  Beta  testing,  testers  recorded  the  time  they  spent  studying  the  various  course 
components.  The  CBT  study  time  was  computed  based  on  the  average  actual  time  spent  by  Beta 
testers  studying  the  CBT  material  and  completing  the  exams.  Based  on  student  experience  and 
proficiency  level,  the  amount  of  time  needed  to  study  doctrine  to  ensure  entry  level  skills  for  a 
course  will  vary  significantly.  This  chart  does  not  include  any  time  for  reading  doctrinal 
material. 

STUDY  TIME  (HOURS) 


COURSE 

HOURS 

Common  Core 

18.64 

Training  Develoner 

5.34 _ 

CRT  Author 

19.42 

TOTAL _ 

43.4ft 

2.  The  following  chart  shows  the  number  of  lessons  contained  in  the  BSTS II  courses. 


NUMBER  OF  LESSONS 


1  COURSE 

#  Lessons 

12 _ 

4 

CRT  Author 

_ 14 _ 

TOTAL _ 

Kffl  ■ 

3.  Each  of  the  subjects  within  each  of  the  courses  contains  a  pretest  which  the  student  must 
complete  in  order  to  receive  credit  for  the  course  within  the  training  management  system  and  be 
allowed  to  take  the  lesson  exams.  A  post  test  is  provided  at  the  end  of  each  subject.  Scores  in 
the  below  chart  were  computed  by  averaging  all  subject  test  scores,  for  all  Beta  testers,  for  all 
Subjects  within  the  designated  course.  The  “improvement”  is  the  difference  between  the  pretest 
and  post  test  scores. 


PRETEST/POST  TEST  IMPROVEMENT  (%) 


COURSE 

PRETEST 

POST  TEST 

IMPROVEMENT 

Common  Coro 

82 

94 _ _ 

12 _ _ 

63 

92 _ 

29 _ 

N/A 

89 _ 

N/A _ 

73 _ 

92 _ 

19 _ 

(N/A  =  Not  applicable) 
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4.  Every  lesson  in  every  subject  of  every  course  contains  a  lesson  exam  that  is  CBT-based. 
Scores  in  this  chart  were  computed  by  averaging  all  lesson  exam  scores,  for  all  Beta  testers,  for 
all  lessons  within  the  designated  course. 

LESSON  EXAM  SCORES 


COURSE 

EXAM  SCORES 

Common  Core 

_ 20 _ 

Training  Develoner 

_ 23 _ 

CRT  Author 

_ 85 _ 

AVERAGE _ 

_ 82 _ 

5.  A  CBT-based,  comprehensive  exam  is  provided  at  the  end  of  the  Brigade  Common  Core 
course.  The  average  comprehensive  exam  score  for  all  Beta  testers  within  the  course  was  82%. 

6.  At  the  completion  of  each  Subject  for  each  course,  the  students  were  asked  to  rank  order  the 
course  components  as  to  their  educational/training  value.  The  columns  in  the  chart  below 
represent  the  course  components: 

-Workbook:  Student  Workbook  introduction. 

-Job  Aids:  Checklists  and  guides  contained  in  the  back  of  the  Student  Workbook,  that  can 
be  reproduced  and  used  by  staff  officers  to  assist  them  in  the  performance  of  their  duties. 

-References:  Doctrinal  publications  used  as  references  for  each  of  the  lessons. 

-CBT:  Computer-based  training  material  (lessons,  tutorials  and  practical  exercises). 

-Exercises:  Subject,  lesson,  and  COMPS  exams. 

The  numbers  in  this  chart  were  computed  by  averaging  the  ratings  based  on  the  assessment  by  all 
Beta  testers  within  the  designated  course.  In  this  chart  “1”  means  the  most  valuable  and  5 
means  the  least  valuable. 


STUDENT  RANKINGS  OF  COURSE  COMPONENTS 


COURSE 

WORKBOOK 

JOB  AIDS 

REFERENCE 

CBT 

■roasiaiKia 

rnmirinn  Cnrp 

4 

3 

5 _ 

1 _ 

_ 2 _ 

4 

3 

5 _ 

1 _ 1 

2 _ 

CRT  Author _ 

3 _ 

4 _ 

5 _ 

1 _ 

2 _ 
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